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/ . The study reported here extends the investigation 
bequn by pi a get of content effects to laboratory and naturalistic 
tasks. I^the accuracy and completeness of the sub jects*'. 0xpec^^ 
variable^ is related *o conjt^nt* then pfirfbraance on laboratory and / 
naturalistic tasks should differ systematically. Ihe subjects* 
expecti^d variables were measured by having sub jects name variables ^ 
and by/a aroup survey. Use of the controlling variables strategy wrs" 
measureid bv four guestions about constructing , criticiasing, planning, 
and ahal-yzing experimental results. Subiecta of different ages and 
socloeconmic groups were used. Results revealed a strong influence of. 
cont*ent .on reasoning perfo;rmance. Contrary to Piagait's Writings, " 
co-n^ent effects werte systematic, in addition, by choosing physics 
problems and questions which deemphasized content, Piaget avoided 
ajiny content influences. Thus, it is sugcrested that educational *^ 
implications oenerated from Piagetian theory nay ba incomplete^: 
bontent effects require more emphasis. (Ruthor/QK) 
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* 

Porm*X rea«onlnp haa baen etudlad aKtenNtvely with tnaUa that; huva 
phyalca or laboratory content—thia rasaarch Invafltlgataa tha affect of con- 
tent on formal reasoning. Ninety, thirteen, fifteen, ond tiovontuen-year-oldo 
received both laboratory and naturalistic content taBks which required ability 
tC) control variables. Expectations about the variables In each taak ware 
measured. Results revealed that 8% to 15% of the variance In performanc« was 
as80clat;ed with task intent. Content effects were shown to reflect expec- 
tatlons about task variables. ' ^ ■ 
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Iff It Vortoni II U*i Not Fhytlfiir 

' ' ' ' ' 

[h« InflutTjiaii of Uboratory iind NufcwF^listAo 
^ ^Contmnt on Formal ll(iia«anlni|), 



. Formal rtaaonlttg has baan atwdiad prfdominataly with tattka that hava 
phyfllea (or laboratc^cy) contanti raiding th« <|ue8tlons la it formal if |t'a not* 
phyelca? Inhaldar aU Piagat's (1958) wonumantal work on formal raliaonln9 focuaed 
on^tha atratagiaa' rajiulrad for formal roaaonlngt auch aa tha strategy for 'con- 
trolling variabloa, jiot on tha tapk content. In thia- paper , 'we examine how ^' 
laboratory and naturalistic content influence the appllcati^A of formal Teaaon- ^ 
i«^g atrategiea. \ ^ 

Inhelder and Pijget (1958) hypothesized that content would unsystematically 
influence performance!; their research denfcnstrated l^||p| concern for content. 
First, their fcaiaks came predominately from physics — they did not compare tasks 
from different content:! domains. Second, they confounded strategy Wth content, 
•for example, using bnd content (bending roda^ to measure the c^itroliying variables 
strategy and another (palance beam) to measure the proportional reasoning strategy. 

0 

Piaget has written very little about content^ Indicating his limited concern. 
In 197|, he discussed "resistances" due to content which influence performance. 
He said that effects of resistances were easy to explain after they had Jccured 
but hard to predict. Piaget believed that content Idlosyncratically Influences 
performance. .One Idiosyijcratic influence described by Piaget is expertise: To 
explain why formal reasoning was less prevalent than he hypothesized, Piaget 
(1972) cited the effect of expert content knowledge. Piaget said Individuals 

were more likely to reason formally In their area of expertise than In other 

\ . \ 

areas. For example, an auto mechanic might display formal reasoning when diagnos- 
ing a defect In an avto but not when diagnosing a child's allergy. 
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In tonfirfit; to Piaget, who did not ire$a*irch cont^af aad t^hougKt content 

pevf6irm«nci«, For any taak, th«j avibjoc^ ©Kpectf thai: c.evt;«l,n''varUlvleH need to 
be con«ld«r«di w© call t jewo tho "«ubj«>ct:ir' «xp«ct:«ul vttviableiC-'. Tlw vrtvlttblwa 

• . ■ - ■■ K 

that the. exparlnvencer aelecta l^ot a task (the "exp^rXmantora' (comprobenaivo var- 
labl^a")- luax differ f romt the aijibjecta' expected vartablea. .Wo. bypatboBlzedYthat 
content will infXuonco £^)rml roaaoivlng task perfornumceybac^UM'^, for a (flveu 
content, subjectii hajrd piirtlcular exp^ctatlouH regarding tho va't^abloa* 

>8yateraattA dlfforen|:e8 between; the aubject'a expected viiri^abloa and the ex-^ 

. * 1 . 

perlmenter's qompreJienaiye variables can Influence performance in the following / 

■■ it'- ' ^ 

ways: Subjects may consider fewer Variables thart the fcxperiraenCjpr, Expectations 

* ... iJ^ 

may also be inaccurate; for example, moat [people inaccurately expect tl^ weight 

of the bob to influence the oscillation of th^ penduium because weight influences 
other phenomena* {I 

While Piaget only studied tasksf^ith laboratory or physics corl^ent, such as 
Bending Rods ^ we ||l^esti'gated contei^t effects by comparing a variety* of labpratory 
and naturalistic tasks. Naturalistic tasks have content' from frequently encoun-v 
tered situations such as determining which is the best toothpaste to buy or how 
to get the best mileage. In the naturalistic tasks a^in the^ laboratory tasks, 
we asked subjects about; the design of controlled experiments.' ^ , 

'We hypothesized that' the completeness and accuracy of the sut^ect's ' expected 
variables would be influenced by content. Specifically, we anticipated that, the 
subjects' expected variables for laboratory tasks would differ frfim the subjects' 
expected variables for naturalistic tasks. We planned to demonstrate systematic 
content effects by relating content d^lfferences in the subjects' expected variables 
to content differs ces, in reasoning. ' 9-' • 

I - ■ 



■ ' '■" '\ . ' '. .. : ' , ' , 

t;h«i controlling vMrUniae iit:rb(:§gy< (Usiing tXm aontrol.lina vnTiabXew utrnt^gy 

, . ^ ' ' % '*}T * * 

ot)i«iw «r« k^apt: cha aiiiwO 8wb;J#ct6* u«#\)^ cout:vo,ll Ing v*i)f UblcJi*i «t:rnt;agy wan 

\\y quaatlowH which li>voXvad contitruct Ingv crUiBisslngi planning, and 

analyaslng axperlmanta. PraArXoiia, raaaarcU (©•g, , I4nn, 1978) haa ahown that thajia 

. quaatlona ravaal dlffarant ^lapatsta of atratagy knowXadga** In^Xhiii atudy> qiiaa- 

l;;lon«. about tho controlling variablaa atratagy ware «alactad to ir a veal tha influ- 

aupa of tha aubjecta* axpqctod v^rlablea^ oq sti^ategy uaa. r 

Prej^lmlnary Studies of Contfent Effecta , ^'^ 

iTwo prollminfiry studies of content effects In f<itinal. reasoning lay the groundr 

work for the hypotheses lnvSstltt«lited in the pre^aaht^paper* Vo'^^Lnveetlgf^te^ ^iaget's 

' . ■ ' ' ■ . A ( ' 

(1972) statements that people are nioi:^ llkMy. t</ Mason itorraally In, their area of 
expertise,. Pulps and Linn (1978) studied two- ^ro.u^^ of "^eVent^h graders who had 
i different levels of expertiscv for controlliag v^riablSM^^asks with fishing and 
;iJU^6ra^Kl?y science' content. One grqup liVed on a river delta ^p^nd often went fish- 

'••.jfLnftr'^^Thef other^^ enrolled in an d^cperlential .jscienqe prog^-ara, mak,^au^|ic 

f ^ Jl r ^ fj^'' ■ * ^ " ^ <^ ^ ■.' ' ' ^ 

hypothesized^ Ahat tn^ subjectsiji expected variables wouy be most ^ttcurate and 



liors 

and 

. 1»k>firt ^completfe ii^ their area of expertise and would lead t6 siiperior performance 
by' experts. They found that ^bjects (Jerfdy^ed better qn controlling variabolis 
tasks in their area of expertise and performed\equivalently ,Gn a neutral content 



task where neither grot^^^^^i^xpert. Prelimlroi^ evidence for a v^elaJfe^onship 
tween* si/bjects' expected varl^ibles,^ and subjects V performance was found. These 
results suggest that expertise influences the ability to control variables. 

iTinn and Swiney (iJote 1) in' a second^ study of content effects with elevenMh 



atloiiishlp' het\ 
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graders, investigated: the relatiorishlp' between the subject's expected variabl^^, 
and tjie variables the subject aetualj^y utilized, on a labb^tory controlling 



C«ik. Th«y measured the aubJactU expected v#ri«blee., by Iwylng aubjaats bx«mine i 
th# taak' appuvntim mui' utuw th»» v«rt«bl.i8 they exp«qted would laf^ueupa the out^ 
<cow«. 8ubjiic(f« were pi'isaiiraed to mm th*ly #«piaat«»d vMriablea. auh.ttiuta ware thpiv 'i 
told th« «xperliiwater'« t;om|)irfcilwi\BJiv« v«cl«bl*«s. 8ub,1atU;t=t ilemoiiHtvtticid ritrti t:h«»y 
uelllK«d a vin"l<»ble by correctly ctrntrrolltng or Invaatlgrttlna it In «n «xpurlnwiU;r<, 
Compnrlaon of the aublauta' exi»oict«d varlablea and tho aubjacta' utillisa«l vai-~ \ ^ 
l«bl«« rev««lad that aub;)ep,t8 utIllKod thair expected varlabUfl alBalf loantly more 
often than otlmr variablea. Subjacta appitfir^d to auawtsr th« cout tolling vat'tftblttH 
quoatlona aa if only thtstr expuetud varlablea waro important aubJiHUa Igi^oy'*^^! 
• the experimenter 'a hypothetical varlablua, * . ^ 

The Proaent Study 

The study reported In this paiier ^ixtenda. the lnve«tlgatlon of content ef fecta 
to laboraftory an<J naturalistic taaka. If the accuracy and completeness of the • 
subjects' expected variables iV related to content, then performance on laboratory 
and naturalistic tasks should differ systematically. The subjects expected var- 
iables were measured by having subjects name variables and by a group survey called 
What Is Your Opinion ? Use of the controlling variables strategy was measured by 
four questions about constructing, criticizing, planning, and analyzing experi- 
mental results. In contrast to earlier studies which used a single age and 
socioeconomic group, this study used three ages (13, 15,^ and 17) and three schools 
differing in socioeconomic status. ' ^ * 
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• ' • . .... .. . " ' ^ 

Hlniftty 13, jlS, iind 17 y»«p olds In fthfte dlfftifine ittheol dlacrlufcs (ttn p«f 

• ««« p«v (llsciriat) p«ifci.olp«t:ad. Four wubjteen mavad during th«» itwdy Uavlng a 
final HAmplf of 86, Th« dlatrleta repr«ii©ntad thra* «oclo«aanoiutc scacua (889) 
groupat Upper middle el«««, middle claaa, and lower middle ol«ia, The upper 
.middle cUuh dletrict wen euburbant familiea owned cheir own honiee end many eeu« 
dents drove their own. care to ichool. The middle claee district waa urbani 
,famllie« lived in single family dwellings, apartments, and condominiuroa'. The lower 
middle claee dlatrict waa aaroiruralj families lived in small single family dwel- 
lings, apartraentB, and condominiums, and many adults were employed in adjacent 
factorlea. Quality of the aclence programa In the three dlatrlcta correlated 
directly with SES* 

The What la Your Opinion ? survey of taek expectations waa administered to 

a larger sample (N « 900) of which the subjects in this study were a randomly 

■ - / 

selected subsample* 

■■ * 

Controlling Variables Tagks 

Six naturalistic and four laboratory tasks were devised. The laboratory tasks 
had apparatus but the naturalistic tasks did not.. Formats to communicate natural- 
istic tasks were pilot tested and three were selected. Some or all of the five 
questions^; (con3tructing > criticizing , planning , analyzing / naming) were asked for 
each task. Scoring was pass/fail, although for some questions, additional record 
was made of specific responses. Table 1 summarizes the format used, t>pe of ques- 
tions asked, and number of questions asked for each task. Tasks were administered 
in two forty-minute Interviews; in the first interview, the order of the tasks 
was systematically varied across subjects; in the second interview, order was con- 
stant. * 

ERIC 5 . 



Met Ft)ll(IW«ii 

I. a«n4tn« Rodi, a laiik ndiiptttd tmm inhiXdif «»id Pl«|it; and ilwwfcVftb^ii 

\by Linn »nd Hwint*y (Nnfce I), «llowi lnvtit4g«t;ton of cUf* b«n\d«hUiiy •ti? vrnt**, 
Ti»« riv* v«rl«bl«»*i i<ir«i the uusterUl of the tod, the tl»lt?ku««»M ot the lod, the 
»ha|()« of thu rod, th» itingtrt ojf |;hi» rod, ^Ad l;h«k i»U*» of thti W0laht: huug from th^ 
rod#\ 

2e Sprlugtt, 4 tMk d^ncrtbed by Liivii ttud Rlct* <l*)79), 1« purMll^l to Hm\dlua 

\ 

Rodii, bu^ »ubl©ct:ii luvudtlgnta t\m ttxpimnton of tiprlngw, V«rl«bUM nr^; tU^ 

\ 

matarlal c^f the spring, th« cro«iB-«actlon of the iiiirlng, th© Umfecih of tho spring, 
the thlckneiva of tha wire, and tha waight hung, from tho aprlng, 

3, Splnlnilng Wha©Xa,'a taak adaptad by Caao (197A) from Inhaldor i*nd Ptaget 
(1958), allowa Invoatlgatlon of centrifugal force. Sptmroa avo placad on a wheol 
which can be rotatad; th© quaatlon to be anawonul 1« which «phero will fly off 
the wheel first. The variables are: the size of the sphere, the material of the 
sphere, the size of the hole In which t^he sphere rests, and the distance of the 
hole from the center of the wheel. 

4. Runway, allows Investigation of linear forces on a toy sled traveling 
down a ramp. Factors Influencing the distance travelled by the sled which Is 
pushed by a marble released from the top of the ramp are Investigated, Variables 
Include: the type of passengers^th<fc numbe*r of passengers, the type of sled, 
the size of the, marble used* to ^pMli the sled, the weight of the marb*le, and the 
height of release of -the marble. 



<i* I** Antics* «iib1«vifli iiiv«e^t igrttci uuiMVrt tnrhi**m*liig ihc» g^** milci*ni«i 4 vnr 

ut ^aciMiiito 11004 V 

UhUM ttr^: ih** tyft^ of biilti lotuutuu^ th^ |ut4««ot«i ot ahMt»iu^^ or u buhbtit , 1 110 
t:yn« of aiul this prutttJiu'ti or wbtiuiuu of n rati to. 

H, in Poiiiulei Awiiy. ^objwiitw luvt^HMH<tt« ffiifotM lof hitiOi' 1 uh I Tbcs 

VAtlMbUd *u«j tb« typti of huxl ^rtrmn, tbo typo iif ^Hdii lnii, I bts iiukuhU tit Mlc^tip, 

r 

th« nmoiiiU of w«6^r roneiumoa \hu: dny , ami thu uumbi^r t)f mtial« aattfo luir day. 

l^.?*Hl**,» ttliowa lovoHt lgaf Ion of this wtilloiuiHH of chitluui oomluH ^^^*f 
wmihtng machlnu* Vnr labloH aro: brand of aoap, prtJiMoneu o\ abaoiuo of bUwu h, 
t\w wator tdmperutiut; y thu pr^riooci^ or abaotuo of fabrlr Moftouor, ami ilio lypo 
of wiiHhlng imichino ufuuK 

iO* Runaromvcl 1b a natural iHt^>t1uik about runnliij;, Hptuul. Vnr lab U»h art*: 
running location, running surface, clothing worn, amount of Ble^p, diet, time of 
day when running, and type of shoes worn, - \ ^ 

Questions 

The five questions In the controlling variables tasks are described In this 
section. The const^ructlng , crit icizing , planning , and anal yzing questions measured 
reasoning about the controlling variables strategy. Two of these questions, con- 
structing and criticizing , were combined to form a controlling score for data 
analysis. The fifth question, riaming, measured the subjects' expected variables. 



|:m4 4 4if f *4r«tlu ti tii huw Ca^ •Im^m ttia t^a ticiuipV* 4it4 w»rcs c^x^Uciit **Wliy *IM yMM 

^li |H»tiit loii oui ( liiilt i hdy haft t liui i<i I 1 1::*! »uiii*i v4t l iiUl«tt ill h4il liiv«i44ng«ii «ia 
ii I' I C3 ( ' t V I I a i) I u , 

UMOoutHuUcid «5HjHithmnU. Sul^jiufn \u\'UiLu\ It t liny lihnMfir*! t ho iMuliMuiiL 

1* l*|Mim^ nteArtfUiiinl lu)w huhjtu Wiviihi invr.U I k^M w tuu \\ ot ( lu^ tixptn I mcnU «r* « 

trHjiiai Imtnit ti would ytui <l<i lo Mod tud wljcU Imu iuuU oi { Ut^^u^' iUiuyt^ti (iH)liitluH to 
ll.Ht of irK|)(H tiiRMU (U Si t <MU[M t^houM I vo ViiilMhhvO *u t MM 1 I y mnk«ui n dlrrtMtMir«j in 

^ 

how fiir (hp SprluHfi oxpaiul?'* Stihlt/i lri pa * w d II tlioy luuduru'd o\u^ or mon^ am 
trolled c*xpi»r IummU H . 

^* A^yilyAL'ia "^^^^^'^^ ahlUty to ctlUcfvU^ an oxperluuMU whc»n «huwu,lfu» 

result but not told the proceduro. For .example. In Bending Rods, two roda painted 
different colors and made of dlflertMit nuitei tain woro placed at unequal lengths 
protruding from a wooden stand. Hanging equal weightji on the rods, the experi- 
menter demonstrated that the short rcul hem more than the long rod and asked, 
"Is this a good experiment to prove tliat short roda bend more than long rodn? 
Why?" and "Do you have any questions to a'^k about the experiment?" The length 
variable Is called the alleged causal variable In this problem because the 
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till) 4|iici«ii Ion *i>»vci|U Hicii ocvak^*! luiioutlf*! i au^i i»Mi^ fiVr chd tciiM>n*-fj r^xik 

IHiMAiB iwll><»|pa. it^Ot *»i.aili|a«i, rui CUM «anj»i*t ti*M»lM hIhUH { Uia utr^ t ;:^ f | V|*c ' 

of ||<it»aM)ici iiii iMttai*4# i»i»h)*i» lo Wf:i»« liiia t hat the twu i 4*1 4i iv^u ,>v4ir 4 h» 

Away, thtt iiU0ivi^W€a> iii^Ui^il, ** I f you U^d lu l.i£*fe w«5|i|$hi, mhai wuul4 yiUi iltt 4ir 

f«»*tsril ly lluiti yiHi itu now/ What uU^tit sniutctutm 7 What alacif** lu natuUtin 

> 

might imUti n t|! r roMulu !a huw tm 4«>wn i ht^t*c» ii«Mtr4 hcaiut. . ,Whr«i taUca/** All v*ir 
(nbUni In b*hi>irtti>iy f ^inkfi wt» 1 tv. v f n.ih 1 <i . Tht* f^vy^it* u:u% tlu^. numlu*r of Vj» i f I tsji 
iiiiHafiii^ For «l l t£ii«k«, ^itttu tht* Nnmln^ ^pi^-fi t ! un , u^t^ n v^%i%i I tut 

Bl£%J 9. 1 ^'il^l^L^J^! ? ^ ^ ^lil ^ ^ ^ 

Four farmats were u^ed, one for each controlling question ais shown tn Tibbie U 

1. The Oi>en format, uatui for Spljinlng Wht^eU and Hunwav, allowed all h^vVls 
of each varlahlV^ to he combined. That * la, for Spinning Wheels *largt- npher<v 
could be placed in any nizt^ holo at any dlatance from the centt^r of ^it^whoi»l. 

2. The f ljcod> format, used for BondinK Rodn, Springs, Auto, and Toothpaste, 
limited the possible combinations of variables. For example, there were eight 
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springs and eight autos (with their characteristics described) so some combinations 

■ ■ - ^ ' ■ ? ; 

of variables were omitted. Thus, in fixed format it wp not possil?le to investi- 
gate the material of the spring using short springs with thin wire because both 
materials \«jg^e not available f^r^ort springs with thin wire. Fixed format was 
thought to be more dif f icult than open format because experiments muat^ be designed 
ta £l^t the available choices, . 

3. The Record sheet format, use^ for' Soaps, Pounds Away, arid Fishing, allowed 
each condition of all variables to be combined, similar ..to the open format. Natu- 
ralistic tasks did not have pbjects to manipulate but the record sheet represented 
the objects pictorially. * (e.g. rather than actual fishing rods, there. were pic- 
tiJres of fishing rods.) Subjects indicated how they would design each experimental 
trial on the sheet. Thus, for Soaps , subjects indicated how they would wash two 
loads of equally soiled clothes to determine whether bleach makes a difference. ^ 
fhey Indicated which soap they would use for the first load and which they would 
use for^. the second load, etc. 

A. A unique format, similar to the open format, was used for Runaround. 
Rather than a record sheet, a chart with the alternative conditions of each variable 
was used; experimental trials were indicated with red and yellow chips. The 
subject designed two runs (on different days) to find out if one variable, e.g., 
sleep, influenced running time. The subject chose a condition for each variable 
for each trial. The successfxil subject changed only one variable at a time. 

Group Measure to Elicit Subjects^ Expected Variables • 

A paper-and-pencil group-administered survey, "What is Your Opinion?" assessed 
expectations about each variable in each task. Two questions were asked:* (1) 
Which condition of the variable do you think will have the greatest effect on the 
outcome, e,g., will the heavy or light weight make the spring expand the most? and 
(2) How much difference do you think the variable will make?: a big difference, 
a little difference, no difference, or don't know. 
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Results and Dl8CU88 lon 
: — — \ — 



Responses were analyzed to reveal content effects on reasoning. Contents / 
employed were laboratory and naturalistic. Questions guiding the analysis were: / 

1. What Is the Interaction between measures of expected variables and lab- 4^ ' 
oratory versus naturalistic, content? ' ^ 

* ■ * . . y/ 

2. What Is the Interaction between each of the four controlling variables 
-questions and laboratory versus naturalistic content? 

♦* • • ' ^ " 

3. What is the Interaction between the three task formats and laboratory 
versus naturalistic content? 

4. Are age, sex, or school related to performance? 

Interactions of, the subjects' exp ected variables and laboratorv v eraufl naturalistic 
cont ent ■ ~~- ^ / 

' ■ ^ y _ ■ 

"^^^ What .Is Your Opinion? survey measured both the accuracy arid the confidence 
ofiSthe subjects' expected variables. Accuracy of expectations was measured by ask- 
ing subjects how each of the experimenter's comprehensive yariables would effect 
.the outcome, e.g., do he£vy or light weights make rods bend more? For laboratory 
tasks 77% of the answers were correct and for naturalistic tasks 81% of the answers 
were correct. Indicating no significant differences In accuracy of expectations. 
Confidence of expectations was measured by asking whether each of the experimenter's 
comprehensive variables would make a "big difference^' In the outcome. Subjects 
eait)ected laboratory variables to m^ke a "big difference" (65% of the time) signifi- 
cantly more often than for naturalistic variables (47% of the time): (t^ = 8.21, 
p < ,01). Thus, subjects had stronger expectations about variables for laboratory 
than for naturalistic tasks. 

The subjects' expected variables for each task were measured In the Interview 
by the naming question. Subjects had significantly more expected variables for lab- 
oratory than for naturalistic tasks (Table 1), consistent with the stronger expec- 
tations mentioned above. The number of subjects' expected variables on one task 
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did not correlate With the number on other tasks to a high degree as reflected In 
the low reliability for naming , - 

Questions abou^ the controlling variables strategy and ^boratory versus naturalistic 
content 

Interactions 'of the* four controlling variables- questions, constructing and 
criticizing , were combined because they correlated highly (•f2), repllc|^hg previous 
studies (eVg., Linii & Rice,' 1979; Linn, Pulos, & Cans, in press). Together they 
form what we call the controlling score. 

. Content effects for controlling , planning , and analyzing questions were assessed 
using average scores for each laboratory and naturalistic task. Means, standard . 
deviations, and reliabilities are given in Table 2 for each content. Reliabilities 
are generally high; slightly higher standard deviations for naturalistic control- 
ling tasks reflect a slight celling effect for laboratory controlling. 

Recall that this study Included two separate interviews for each subject. 
Analysis was done separatiely for each Interview and for both interviews combined* 
No interview effects wiere found. 

Content effects were analyzed using a repeated measures analysis of variance. 
Results (Table 3) are discussed for controlling , planning , and analyzing questions 
below. Since male/female differences were never significant, this variable was 
dropped from further analyses. . 




Insert Tables 2 and 3 about here 
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Content 'effects on Controlling questions . Recall that controlling includes 
construction questions and critic^-zing questions. Subjects must design or criticize \ 
experiments.' ^ 

^We found that dontent influenced controlling. Laboratory tasks were easier 
than naturalistic tasks^ as- shown in Table 2. Success was high (65 to 80% correct 
responses)* Using analysis of variance (Table 3) we found that more than 12% of 
the variance in ^controlling was accounted for by\con,tent. 

Controlling performance reflects differences in the subjects' expected variables: 
they control more laboratory than naturalistic variables and also expect more lab- 
oratory* than naturalistic variables to be- imjportant. ^ 

. Explanations of errors qn controlling varied for 'the two contents. Unsuccess- ^ 

' t» ■ ' . ' ■ ■ _ . ^ ' 

ful subjects were asked why a specific variably in their experiment was not cbri- . 

trolled. If this variable was not one of their expected variables we anticipated 

^ ' ^ I- • : 

the explanation would be "I don't think that variable influences theVgutcome". Con- 

/• . . ■ .1 

sistent with the smaller number of expected variables for naturalistic- content , / 

these responses were twice as frequent for naturalistic (10%) than for laboratory 
tasks (5%), a significant difference (t 2.93, p <.01). Thus content effects for 
controlling reflect differences in subjects' expected variables: subjects con- 
trolled more laboratory than naturalistic variables. When they failed naturalistic 
tasks they were likely to claim that the variable omitted did -not influence the 
outcome, presumably because it was not one of their expected variables. 

Content effects on planning questions . Recall that planning questions request 
the s.ubject to plan a series of experiments. As expected, planning is more difficult 
than .controlling: 28% success for naturalistic, 53% for laboratory. 

Content effects for planning replicated findings for controlling. In repeated 
measures analysis of variance, content accounted for 10% of the variance in plan- 
ning, with laboratory content easier than naturalistic content (Table 3) . As for 



o 16 
ERIC 



controlling/ content effects reflect differences In the subjects* expected varlableS^: 

'' ' A ■ 

\^ubjects control more laboratory than naturalistic variables. 

Content effects on analyzing queistlgns * As described above, analyzing ques- 
tions require subjects to criticize experiments where they are told an alleged I 
causal vaAable but are not told whether other varlablear were controlled. Analysis 
Is more difficult than controlling but easier than planning (52% success on analyz- 

■* . * ' ' ' 

Ing compared to 72% success on controlling; ^nd 41% success on planning), • 
. Crfntent effects for analysing were found: ^ naturalistic tasks were slightly 

; - ' ^ ■ ,.v ■ 

eaaler than laboratory tasks (Table. 3). Content effects' for analyzing questions 
were consistent with differences In the subjects*, expected "variables. Successful 

responders could ques^:±on the ^alleged causal relationship, saying: "That experi- 

' ^ , ' y ■ ■ 

meQt^ 1^ bad because some cJbh^rj variable probably caused the 'outcome"^. Subjects had 
to expect that some other variable caused the^utcome in order £o question the 
causal relationship. Subjects , who questioned the alleg^ad causal relationship p rob-' 
ably did so becatj^e the variable alleged to cause tl\e outcome was ndt one of their 
expected variables. Naturalistic tasks had fewer expected variables than labora- 
tory tasks. In fact, alleged variables in analyzing questions were subjects expected 
variables, 92% of the 'time for laboratory compared. to 49% of the time for naturalistic 
analyzing questions (t = 8168, p<.01). Therefore, questioning the alleged causal 
relationship should have been more common for naturalistic than for laboratory 
tasks, becayse the alleged variable was less likely to be one of the subject's , 
expected variables. ' , • 

We verified the impact of the subj^^ts' expected variables on performance by 
examining reasons^^for^ correct responses. Two types of correct responses were, pos- 
sible. One response was to question the alleged causal variable (e,g, "It may not 
be the length of the rod like you said, but the material that caused the effect"). 
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Another response was to cite a rule about controlling, saying: "I don't know If 
all the other prossijjle Influences wejre the same"* Questioning the alleged causal 
variable was almost twice as likely for naturalistic analyzing questions (43%) as 
for labbratory (26%) analyzing questions. This difference was significant (t = 4.3, 
p <.dl). In cont^rast, for bot^l^ contents, citing^ the rule "everything else should 
be the same" .was equally likely and infrequent (5% of response^^><^ Thus subjects 
criticize analyzing qy^stion repults pore frequently .if the<^lleged causal variable 

t.was not "one of^4ieir expected variable^. | - 

What happened; when the subjects' expected variables were not -available to ex- 
plain the analyjsis question outcome? On the controlled analysis question, subjects 
were told that expected variables could not have caused the outcome reportect in the 

« question. -Thus, likely answers to the question were eliminated. On the controlled 

: ^ ' • ■ ■ - ■ ^ ' \ ^ ■ 

analyzing question, subjects were dramatically less successful (19% success) than^ 
; on^ the regular question (40^success])[,.' Few success^yl respondearWsuggestfed that 

^V/ * • r- ' ... ^' . r ' ... -W^'. ■ f 

Another variable; caused the outcome since most alternative cause§v)jwere eliminated 
by the questions. Successful responders, instead, stated'the controlling variables 
rule. The rule was morq likely to be usec3|j|fc, for this specific question (14%) than 
for any other analyzing, constructing br criticizing- question. It appears that 
* the rule is available but is not usually the first response generated. 

Relationship* Between Reasoning' Questions ^ 

Scores for laboratory and naturalistic versions of each question were reasonably 
^related: Controlling ^\75; Planning « .73; Analyzing = .56. The relationship 
among controlling , planning , and analyzing was generally high (average correla- 
tion a .65). These correlations suggest a common factor in all tjie reasoning ques- 
tions, y ' . 




O ^ } [ . ■ ' 

Interactions between task format and laboratory versus naturalistic content 

We hypothesized that fixed format would be ^ the most difficult, and the results 
y hear thl6 out (Tables 4 & 5) . The fixed foirmat requires keeping more information 
In^mind^than does the open or record forinat: variables cannot be cont;rolled se- 
quentially* In Bending Rods, for exapmle, the subject might have chosen two brass 
rods to investigate =cross-sectlon only to discover that cross section Is confounded 
with thickness for brass rods. In open or -record format subjects can choose con- 
ditions for each variable and always find unconfounded tests. 



Insert tables and 5 about here 
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Fixed format is more difficult; does format interact with content? It appfeiftS^f/* ''i'Vl 
not. Content effects performance on fixed format about the same waV^it effects Hper-*^^^^";^^^^ 

/ ' V •. - . .-. ' \^ -/^^.fi 

^Bprmance on ppen and record formats. ^ . . . 

Effect of sex, !age^ /and school on performance ''-M^, 
, ^ As noted above, there were significant effects for sex so this variable was 7 

dropped from further analysis, j _ ?y ^ ■ • a ^ ^ ^ 

W 'X ^ - ^ ' 

The age related Improvements In performance on all reasoning questions reflected > 

k developmental trend. Age accounted for the most variance In planning and the 

least In the con t ro lllng\ ^ sugges t ing that- planning was more Influenced by a develop- ' 

mental mechanisjoa. W^^nVestlgated three explanations of these ffndlngs: (1) across 

the board acquisition of a formal reason-ii!?g*^ strategy; (2) changes In processing 

capacity as described by de Rlbauplerre and Pascual-Leone (1979); or (3) increases 

In content knowledge. 

\ The first explanation, acquisition of formal reasoning could account for 

Improvement In controlling , planning and analyzing; with age If some subjects at 

each age acquire formal reasoning. How can acquisition of formal reasoning be 

established? We could consider a Object formal if they correctly used the controlling 
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\ ■ ^. • • ' \ . ■ ■• • - . • -ir ' 
strategy at least once but then all subjects be classrfjle^ as formal. We 

N^ould consider a subject at fonna^ if they responded to the planning question 
correctly since planning resembles tlie controlling task used by Inhelder and 
i Tiaget (1958). The planning question required a more complex strategy, keeping 
: track of several experiments; .there were subjects who failed all planning questions, 
suggesting that planning may develop in tl^s ^ raiige. Thus sojne sut>port for an 
increase in formal ' reasoning is. possible if the definition of formal is choseri 
correctly. . ^ f / 

Processing S^pacity explains age related changes in performance by an increase 
in the amount of information . that can be processed. The Increasing' difficulty 
of cpntrolling, analyzing, and planning questions reflects the Increasing nfeed 
for processing -capacity. As discussed fey Linn & Levlne (1978),. analyzing requires 
more processing capacity than controlling because the resu^lts of the an^alyzing 
question must ^ be kept in mind. Plannin^< since it requires designing a series 
'of elcperiments also requires more processing capacity than controlling.. Thus, an 
increase In processing capacity may explain the age related increases in reasoning ' 
performance as well as increases in the difficulty of the reasoning questions. 

Age related increases in content knowledge are reflected in increases in ' ^ ^ 
Naming and therefore in the subjects expected variables. Thus, older subjects ^ ' 
might perform better because they have more accurate expected variables thmT^"^ 
younger subjects. Further evidence for content/knowledge effects comes from the 
relationship between school and reasoning performance: Students in high SES - 
schools outperfoCTed those in low SES schools (Tables 4, 5). ScTiool accounted 
for about 10% of the var/ance in each reasoning question. Increases In content 
knowledge are associated wi,th better performance but the causal role of content 
is uncertain. 

Acquisition of formal reasoning and processing capacity could each explain 
sage effects; their relative importance is difficult to ascertain. Content/ 



knowledge, however, clearly plays a role. Content could have a direct effect or 
Content could Indirectly Influence performance by influencing strategy application. 
Another possible indirect content influence could be a reductioti of the processing 
capacity requirements of a task accompanying increases in content knowledge, " 
In summary, we found that » content influenced reasoning. The subject's ex- 
pected variables influenced perforinance on controlling variables questions. 
Subjects expect to, consider more laboratory task variables than naturalistic task 
variables. . * 

I Performance on control ling and planning .controlling variables questions was 
more I successful for laboratory than for naturalistic tasks. ' Performance on 
analyzing controlling variables questions was more successful for naturalistic 

than ^or laboratory tasks* 

\ ^" ' . ' ^ 

]^ appears that performance on controlling and planning . questions is directly 

ft ' , _ ■ . • 

related to the subjects' expected variables. Subjects tend to consTder^th^r' 

expected variables in designing experiments, and they have mor^ expected variables 

for laboratory than for naturalistic tasks. In contrast, it appears that performance 

on analyzing questions is inversely related to the subjects' expected variables. 

If the alleged causal relationship in. the analyzing question is not about one of 

the subjects' expected variables then the subject (correctly) queries the causal 

relationship. Since the alleged causal relationship for* laboratory tasks is more 

likelyj to be- about a subjects ^ expected variable than it is for naturalistic 

tasks, analysing questions are more difficult for laboratory than for naturalistic 

tasks. 
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\ Implications 

These results reveal the strong Influence of content on reasoijlng performance: 
between 8X and 15% of variance on- the controlling; variables reasonlhg questions 
was associated with content • Contrary to some of Plaget's writings, content effects 
were systematic. In particular, the subjects' expected variables did influence 
• how the problem waS solved. 

At the onset we asked: Is it formal if it's not physics? We^&und that by 
choosing physics problems and questions which deemphasized content Piaget avoided 
many of the content influences. Piaget chose laboratory tasks 'and since most 

V - • ^- ■■ • , ■■• 

laboratory variables are part of the subjects' -expected variables content affects 
' ■ . \ " ' '- 

? accounted for* little variance. Piaget chose questions to assess the controlling 

« • . 

variables strategy which emphasized the procedure for a controlled experiment and 
minimized content. \ci contrast, our analyzing questions required reasoning ^bout 
the procedure for a controlled experiment but alsb required reasoning about the 

experimental outcome. In analyzing questions when the experimental outcome was 

•■i 

emphasized we found that subjects frequently considered the content in their 
response rather than applying the controlling variables > strategy. 

Content effects explain why certain reasoning problems are failed by subjects 
who know the controlling variables strategy: subjects only consider their ex- 
pected variables in applying tlie strategy. ^Thus, subjects apply the strategy 
correctly to their expected variables but do not consider other variables. In , . 

- explaining why they did not control a variable, subjects frequently said, "That 
• • " • ' , / 

variable does not make any .difference," suggesting that the variable was not one 

of 'their expected variables. * 

Content effects clarify research findings of pervasive inaccurate physics 
reasoning. McDennott (Note 2) reports inaccuratte reasoning about acceleration; 
Clement (1979) .reports Inaccurate reasoning about force; Champagne, Klopfer, and 
Anderson (1979) report Inaccurate reasoning about mechanics. If subjects have 
O . no 

ERIC ^ ^ ^ ^ 
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Inaccuratf expected var^bles |hey might rea$on inaccurately because they only 
reason a|>out their expected variables. In the research teported here, subjects 
/considered only their expected variables thereby omitting important variables* 
In other situations by reasoning about their expected variables, subjects might 
Include uninq>ortant variables such as including weight as a variable in the 
oscillation of the pendulum. Thus, studies showing inaccurate physics reasoning, 
may reflect inaccurate expected variables. \ 
Task Characteristics 

Our laboratory and ntlturalistic tasks differ in important respects. These 
differences are largely ecologically valid. Our laboratory tasks involve apparatus, 
represent closed systems, and involve some variables, such as weight and length 
on bending* rods, whose effects are well understood by subjects. In contrast, 
our naturaligtib tasks involve printed and verbal information, represent open 
systems, and involve some variables whose effects are poorly understood by subjects 
likess.the effect of amount of water consumed and type of diet on weight loss. 

Laboratory tasks such as bending rods are usually encountered in science 
classes accompanied by apparatus. Usually the task variables can be manipulated 
by using the apparatus. In contrast , naturalistic tasks such as determining how 
best to lose weight are encountered in a wide range of situations and are not 

necessarily accompanied by apparatus. Usually the task variables can be manipu- 

# ^_ 

lated but may be difficult to systematically t investigate because, they covary 
with other variables. For exMple, weight loss may occur when the subject is 
feeling optimistic and eating less. 

By closed versus open system we mean specified versus unspecified limitations 
on the task variables. Laboratory tasks are often closed systems where only 
C^i^tain variables can be investigated and where the choices about how to conduct 
the investigation may be limited (rods may only have two possible thicknesses); 
In contrast, naturalistic tasks are usually open systems where the possible \ 
» - - ■ , 23 -. •;.4 \ .. 
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variables and choices for those variables seem fairly unlimited. 

We have shovm that subjects' expected variable^ differ for laboratory and 

i 

naturalistic tasks. These tasks differ in important respects. Subsequent research 
could clarify the precise role of apparatus, openness of the system, and under- 
standing of the variables in laboratory and naturalistic task differences. 
Question Characteristics 

By adding the ^ analyzing questions to those traditionally asked about the 
V 

controlling variables strategy we found interactions between content and question. 
Controlling, was easier for laboratory t^sks while analyzing was easier for 
naturalistic tasks. Analyzing is an important question because it more closely 
mirrors what happens in naturalistic problem solving than does controlling or 
planning . Reasoners rarely set up contf oiled experiments to test their ideas 
because they don't have the time or resources. Instead they are likely to analyze 
incomplete information. Even in the controlled analysis question subjects only 
ment:|,on the controlling variables strategy when they cannot generate an alternative 
variable that could explain the outcome. Thus, performance on analyzing may more 
closely resemble naturally occurring problem solving than does performance on 
controlling . 
Expectation's 

Our hypothesis that expectations influence reasoning performance is. supported 

by this research. We found that the completeness of the subjects' expected 

1 

variables influenced performance. How do | these expectations develop and how can 

• "I 

they be modified? * 

Subjects develop expectations from experiences with the variables. They 
successively refine their expectations as new information is encountered. Ex- 
pectations become situation specific only when forced by pervasive evidence. 
For example, in the Pendulum subjects expect weight to influence oscillation rate 
(although weight does not influence oscillation rate) because weight is influential 
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in foost situations (like the Balance Beam). Research has shown that most subjects 
expect weight to influence the oscillation of the Pendulum and are not easily 
.altered to be situation specific by evidence from controlled experiments (Linn, 

1977). ; 

The refinement of expectations could enhance reasoning. Expectations about - * 
laboratory and naturalistic tasks are verifiable but rarely verified. Encouraging 
subjects to verify their expectations might enhance reasoning. Obviously, it is 
Inefficient to verify all expectations so a procedure to aid recognition of 
potentially inaccurate or incomplete expectations might foster reasoning. 

Expectations systematically affect formal reasoning. Subjects iray not 
consider the experimenter's comprehensive variables if they are not among their 
expected variables; Expectations are difficult to modify since they are rarely 
verified and are refined by many experiences. One counter experience does not 
change an expectation. Improvement in formal reasoning may occur when subjects 
verify their expectations. 

Philosopher of science Thomas Kuhn depicts two types of scientific research: 
normal science and revolutionary science. During normal science, research is 
guided by a paradigm or theory which describes the expectations of the researchers. 
Normal scientific research follows the paradigm; researchers gather information 
to augment the J)aradlgm not to change It. In contrast, during the revolutional 
science, there is a paradigm shift. Revoluti^ary science occurs when the previous 
assumptions, ideas, and even problems are giWn up and ^new paradigm emerges. 
During these times, expectations are not universally agreed upon. Revolutions 
Involve questioning of expectations; . there is controversy in the field about the 
Importance of each variable. 

If our tasks can be thought of as exemplars of normal, science and of revo- 
lutionary science, our laboratory tasks resemble normal science while our natural- 
istic tasks resemble revolutionary science. "^For our laboratory tasks, expectations 

^ • 25 
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are strongly held and thpre is consensus about the procedures for gathering 
additional information. In contrast, for our naturalistic tasTtts, there is vari- 
ability in the expectations about the variables, few variables are expected to 
influence the outcome, and procedures for gathering new information are contro- 
versial. Thus our naturalistic tasks may more closely reseinble revolutionary 
scientific Investigations than our laboratory tasks. Naturalistic tasks simulate 

r 

creative scientific research because there is controversy about expectations for 
the variables. 

Classroom instruction litlg^t employ naturalistic rather than laboratory tasks 
to simulate creative scientyHp research and to motivate students to go into 
science, 

\ 

Kuhn points "^ut that rdl^l](P^ons do' not necessarily lead to superior logic 
on the part of the,Vcientists who participate. Of course, revolutions or paradigm 
shifts do result in sdbie sorts of intellectual gains. These gains may be in the 
content or knowledge of the area not in the logic, Siegel (in press) points out 
that revolutions do not imply logical development as Piaget defined it; increases 
in content knowledge do not imply increases in formal reasoning ability, for 
example. However, revolutions do imply changes in expectations about the variables 
in a particular problem. During revolutions, researchers evaluate their expecta- 
tions and change some of them. This progression^ of knowledge may occur in addition 
to the progression' in logic described by Piaget. If reasoners can be taught to 
go through normal and revolutionary periods then instruction which encourages 
revolutions may be helpful. Adolescents have many inaccurate expectations which, 
if investigated, might be altered. Thus, instruction to encourage revolutions 
would encourage verification of expectations. 

Naturalistic tasks are important to insure that science education fosters 
scientific literacy. Instruction using naturalistic situations would enhance 
the relevance of the instructional program. It is essential to provide instruction 
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that helps students evaluate expectations abput naturally occurring problems. #. 

Laboratpiy tasks may not require such evaluations. Since ^t Is apparent thnt 

evaluatldn of expectations la very Important for naturalistic problem solving, 

efforts to choose problems for science Instruction to llJLustrate the role of 

^^xpertAf Ions would enhance scientific literacy. 

These results suggest that educational Implications generated from}Plagetlan 

• » 

theory may be Incomplete. Content effects require more emphasis. Consistent ^ 
with Duckworth's (1979) suggestions for younger children,' variability in performance 
on reasoning problems may not be due so much to lack of developmentaliy based 
strategies as to lack of instruction in how to combine expectations and strategies. 
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1.89 .74 
2.31 '52 
31 



2.74 .69 
2.16 .76 
2.51 .^9 
59 



3.24 .57 

5. 

2.55 .92 
2 . ,96 .^S9 

V:'49; 



3.06 .57 
2.43 .89 

2.81 .59 

I- 



47 



2.81 .87 
2.16 .73 
2.55 .66 
62 



2.70 .55 
2.02 .88 
2.44 .60 
64 



■''Mean percent success, dedtnals omitted 



an nusdier of variables named 



09" 



Tfbla 3^ 

Compurlsolf OH UiborAtory and NaturallBtlo Content 



■p'^"""-—"" 




































- • 




Bf facta 












Contcint 






Age 




School 

1 




significant 
' Intfifactlona 






F 


Sig. 




P 


Slg. 


0.2 




Slg. 




V Slg. 


2 


Cont^rolllng I 


35.64 


<.01 


.14 


6.44 


<.01 


,08 


7,44 


<.01 


.11 


3.06 <.05 


.02^ 


Controlling II 


24. 33 


<.0l 


.12 


2.38 


na 




9.24 


<.01 


.18 


3.80 <.05 


.02"^ 


Controlling l/II 


44.48 


<.01 


.20 


4.43 


<.05 


.02 


8.71 


<.01 


.15 


5.07 <.01 


.03''' 


Controlling 
(Fixed Format 










• 














Only) 


31.42 


<.01 


.12 


4.38 


<.05 


.02 


• 9.27 


<.01 


.16 






Planning 


28.68 


<.01 


.10 


'9.83 


<.01 


.17 


6.32 


<.01 


.11 


none 




Analysing 


14.70 


<.01 


.08 


6.11 


<.01 


.11 


6.61 


<.01 


.12 


none 




Naming 


46.06 




<.01 


.21 


, 7. 13 


<.01 


.13 


2.27 


ns 




none 




. 1 ■ 





School X Question Content 



ERIC 



33 



f. 



ilS 



Fix«d Format 
Open Format 



naturalist ic 
Fixed Format 

Record Format 




3.3ili 

6.73 3.71^ 



Age 



SD 




5.67 3.63 



5.89 3.38 
7.10 3.70 



SD 



8.68 1.47 
9.07 1.75 



7.18 1.07 
7.36 3.73 



School 



Uppar • 



SD 



8.82 1.39 
9.35 1.13 



7.96 2. AO 
8.79 2.20 



Mlddltt 



X 8D 



7.71 2.44 
7.96 2.47 



5.96 3.16 
6.75 3.86 



toweir 



SD 



6.50 2,49 
7.74 2.94 



4., 44 3.42 
4.64 3.72 




\ 



34 



ERIC 



QoiVUfiion Qt fQfmta Within tabamtory 
•n4 Natufillitla Contant 



'm\ ■ 














?:. ■■ / ■ , 


VorntAt 




Age 




1 * 


ichool 


Slgnlficae^t! 




)f Slg. 


2 J, 


Slg. 


2 


F 


Slg. ^ 


F Sig. ^ 


laboratory 

Controlling 

Quaatlona 




• 












Flxad verau0 
Open format 


!!.A7 < .OX 


5.A6 


" ^.Ol 

• 


.10 


5.86 


■<.01 .11 


none 


'Naturalistic 
Controlling 
Qiieatlona 

■* 














t. 
'} 


■ Fixed versus 
Record format 


3.96 <.05 




ns 




10.70 


<.01 .19 


-k. 

none ?i 



only those accounting for one or^ore percent 
of the variance are noted. 



